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ABSTRACT 


PeerEval is a mobile app created for evaluating student presentations. This paper covers the rationale for creating such an 
app, the features of the app, and the benefits for both teachers and students. The details and initial findings of ongoing 
research into the efficacy of PeerEval will be briefly discussed. 


KEYWORDS 


Peer Evaluation, Assessment, Grading Rubric, iOS 


1. INTRODUCTION 


This short paper deals with in-class presentations and a new mobile app that can be used for peer evaluation. 
Japanese students have been called “mobile natives” (Gobel & Kano, 2014) who are more at home using 
mobile devices than desktops and laptops. Previous research (Kano & Gobel, 2013; Lockley, 2011) has 
shown that given the choice many Japanese students will choose completing assignments using smartphones 
rather than computers. Some Japanese universities have tried to accommodate this propensity by offering 
software such as PowerPoint and Word as applications that can be downloaded to their smartphones. 


1.1 Introduction to PeerEval 


PeerEval is a mobile application that allows students to listen to presentations and evaluate them in real time. 
The results are compiled in a database and are then available to the teacher and the students. 

The application was developed in response to the increasing popularity of simultaneous in-class 
presentations. This kind of presentation, since they are given to a small group of people, reduces the stress 
involved in addressing a group, while the peer evaluation promotes greater involvement of the students in the 
presentations to which they are listening (Cote, 2013). 


1.2 Benefits of Small in-class Presentations 


Getting students to practice speaking in class is challenging since they do not possess the skills or confidence 
to speak spontaneously. Short talks allow students to prepare in advance, but whole class presentations can be 
stressful. Small in-class presentations allow students to speak in small groups, thus reducing stress. 

One significant problem with this approach, however, is the fact that it is impossible for the instructor to 
evaluate all presentations when multiple presentations are going on simultaneously. Thus the students’ 
presentations must be evaluated by their peers. With proper planning, this can have an advantageous effect on 
both L2 acquisition and general presentation skills (Hansen & Liu, 2005). Evaluation on paper, however, 
creates considerable paperwork for the instructor, who has to compile the results into a spreadsheet. In 
addition, students are often reluctant to fairly evaluate their peers or give critical comments on paper-based 
rubrics. Using the forum function of a CMS, such as Moodle, poses different problems, since the students can 
see each other’s grades and comments, they are reluctant to grade or comment critically. In addition, grading 
and commenting on Moodle may be asynchronous, thus affecting the reliability of the ratings and comments. 
The PeerEval system was designed to overcome these problems. 
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1.3 Concerning the Reliability of Peer Assessment 


In the field of language teaching, peer assessment has mainly been used as a means of giving students 
feedback on their writing. Some work, however, has been done on peer assessment of speaking. Patri (2002) 
found in a controlled experiment with multiple classes that “peer-assessment was in high agreement with the 
teacher-assessment. This suggests that, in the presence of peer feedback, the students were able to make 
judgments of their peers’ oral presentations comparable to those made by the teacher.” In our situation, 
however, the primary purpose of the assessment is not so much to evaluate the students’ ability so much as to 
encourage them to prepare well and to provide them with multiple chances to speak. Additionally, the 
assessment procedure gives their peers a clear purpose to listen and to compare the speaker’s production 


skills to their own. 


2. PEEREVAL IMPLEMENTATION 


The system consists of two components, a browser-based system for the instructor to create the evaluation 
criteria, to upload the student name list, and to download the results, and the app that the students use for 
their assessments. 


2.1 The Browser-based System: http://peereval.mobi 


Any teacher can use PeerEval with their students. The teacher needs to go to http://peereval.mobi, where are 
two login choices. Teachers may use the system without registration, but they must configure their session, 
conduct their class and download their results within a set time period. Teachers with login-access, however, 
can create multiple rubrics, sessions and classes which remain in the system until the teacher elects to delete 
them. Figure | shows part of the teachers’ page that allows them create or reuse rubrics, set up multiple 
sessions (a “session” is a combination of a class list with a rubric set that can be used for one activity in the 
class), view or download the results. Teachers can choose one of the default rubrics or create their own. A set 
of up to six rubrics can be set up for one session, with a choice of four-point or five-point Likert scales. 


“View resus Dowrioad Delete 
“View resus Downioas__Delete 
“View resuts Dowload Delete 


wre a aka - A 


Figure |. Teacher’s Control Panel 


Red and green clickable icons allow a session to be closed so that students cannot continue to submit 
evaluations after the activity is complete. In cases where the activity takes two or more classes to complete, 
the session may be opened and closed as needed. 
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2.1.1 Output of the Results 


The teacher can show the results instantly on the class screen if s/he is not concerned about student privacy, 
or print them out and supply the students with their own scores which also shows the class averages 
(Figures 2 & 3). 


Figure 2. Complete Session Report Figure 3. Individual Score Report 


The scores can also be output as a standard CSV file which allows the instructor to manipulated the 
results as needed and create grades for the students. A future improvement will allow the instructor to 
stipulate a weighting for each rubric. For example, in a low-level English class, an important aspect of the 
presentations is the students’ preparation so that they speak from memory rather than read their talk from 
notes or from sentences that appear on the slides. Thus that aspect can be given more weight in the final 
grade determination. 


2.2 Using the PeerEval App 


The iOS app is downloadable from the App Store (free of charge). When students activate the app, they log 
in with an access code provided by their instructor and their first name (or handle). Android users and people 
who want to use their PCs to evaluate can use a browser to log in by accessing the PeerEval website 
(http://peereval.mobi) and choosing the “Students” link. 

A screen such as that in Figure 4 then appears with the rubric constructed by the teacher and, when slid to 
the right, the names of all of the students. Touching the name of a student then sets the main screen to 
evaluate that particular student, with the presenter’s name appearing at the top of the screen. All evaluations 
are on a Likert scale. Students are also encouraged to enter a comment in the box provided at the bottom of 
the screen. Each student then submits their evaluation to the server. It is possible, however, for a student to 
return to a previous evaluation during the same session, revise the scores and re-submit. Once a student has 
finished presenting and the peers have input their scores and comments, students can view their scores for 
each rubric, the total score for the presentation, and the class average. 


170 


14th International Conference Mobile Learning 2018 


eee. SoftBank > 18:29 7} 100% Gam + eee. SoftBiank F 18:20 7% 100% G+ 


“o) 


Fuka's Presentation <o} Fuka ae Fuka's Presentation 
Me Pronunciation 
O@OO® mn 1D @@@! 
aa Preparation 
O@O@O® mre 1M @@@! 
(Content) sic Content 
forexorore) ~ [O©OG 
=> iat Organization 
O@OO® ™ |O@@@! 


Your topic Your topic was interesting! 


Total: 14/20 (submit } Total: 1¢ 


Figure 4. Main Screen with Name List of Students When Slid to the Right 


2.3 Student Attitudes towards the PeerEval App 


To informally measure students attitudes towards PeerEval, a twelve item questionnaire was created and 
administered to students who had experience with both paper-based and PeerEval peer evaluation procedures. 
The rubrics for both procedures were similar. The questionnaire was divided into five questions about the 
PeerEval app itself, three questions comparing paper-based and app-based evaluation from the presenter’s 
point of view, and four questions comparing paper and app-based evaluation from the point of view of the 
audience. 

In general, students felt that the app was easy to install and use. They felt that the evaluation system was 
clear and that the comments and feedback would help them prepare more for subsequent presentations. From 
the presenters’ standpoint, the feedback speed was a major plus, and the ability to quickly read peer 
comments was viewed as helpful. One negative point was that presenters felt that students might be paying 
more attention to the app (or their phones) that the presenter. From the point of view of the audience, 
everyone preferred the privacy and feedback speed of the app, and felt that the app gave them the feeling that 
the presentations were more interactive. Students also felt that the app allowed them to evaluate more 
accurately than using paper-based methods. 


3. CONCLUSION 


The PeerEval app has a number of advantages over paper-based or CMS (Moodle) based evaluation. Firstly, 
the interface is simple and easy to use for mobile natives, allowing for faster evaluation and display of 
results. In addition, privacy of the evaluations is greater than Moodle forum or paper-based systems, 
hopefully resulting in more accurate grading and more salient comments. From the teacher’s side, there is far 
less work inputting the data and summarizing the results. 

Table 1 shows a sample of comments from students. Since the comments are anonymous, they tended to 
be quite direct and on point. 
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Table 1. Selected Student Comments 


Session Comment 

it is generally good presentation but I couldn’t understand a little about the content so if you prepared 
little more, that would be great. 

44 hard to follow. too many characters on the slides. 

Good job. Overall good, but some explanation was not totally accurate. 

Slide and organization was good but if you speak more slowly and clearly, that would be better. 

The present ion was slightly long. Maybe you should tell us why you choose this topic more. 

68 Thesis is little bit unclear and not really enough. I am curious how do you broaden your idea. 
References are not enough. You should look for reference of books. 


The preliminary results of comparing PeerEval with paper-based forms of peer evaluation allowed us to 
take student preferences into account. Students generally had a favorable impression of the system, but 
presenters felt audience members were paying less attention to presentations and more attention to their 
phones. This preliminary result suggests that rules and/or procedures should be put into place regarding when 
students are allowed to input their evaluations. 

Finally, since the initial review of the app, the developers have added a student peer evaluation section to 
the PeerEval website, allowing students to evaluate from browsers as well as the iOS app. Browser-based 
peer evaluation functions in exactly the same way as the PeerEval app. This addition helps alleviate the 
problem of having a peer evaluation system on one single platform. 
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